printed circuit board manufacture

ABSTRACT

Method and systems for circuit board manufacture and inspection are provided. A captured image of a circuit board is compared with a target image of a reference circuit board at a similar stage in manufacture. The target image and a captured image are aligned. The captured image is divided into a plurality of image regions. A primary image region within the captured image is selected, and a target image region within the target image is selected and properties of the primary image region and the target image region are compared.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a divisional of U.S. application Ser. No. 12/975,192filed on Dec. 21, 2010; which is a divisional of U.S. application Ser.No. 10/767,216 filed on Jan. 29, 2004, which issued as U.S. Pat. No.7,860,854 on Dec. 28, 2010; which is a continuation of U.S. applicationSer. No. 10/216,550 filed on Aug. 8, 2002, which issued as U.S. Pat. No.6,721,733 on Apr. 13, 2004; which is a divisional of U.S. applicationSer. No. 09/179,541 filed on Oct. 26, 1998, which issued as U.S. Pat.No. 6,463,426 on Oct. 8, 2002, which claims the benefit of U.S.Provisional Patent Application No. 60/063,623 filed on Oct. 27, 1997.

STATEMENTS REGARDING FEDERALLY SPONSORED RESEARCH

This invention was made with government support under Grant No.N00014-95-1-0600 awarded by the U.S. Navy. The government has certainrights in the invention.

BACKGROUND OF THE INVENTION

This invention relates to information search and retrieval systems andmore particularly to search and retrieval systems which utilize in wholeor in part image processing techniques.

As is known in the art, a digital image is an image which may berepresented as an array of pixels with each of the pixels represented bya digital word. Often the array is provided as a two-dimensional arrayof pixels. With the increase in the number of available digitalpictures, the need has arisen for more complete and efficient annotation(attaching identifying labels to images) and indexing (accessingspecific images from the database) systems. Digital image/video databaseannotation and indexing services provide users, such as advertisers,news agencies and magazine publishers with the ability to browsethrough, via queries to an image search system, and retrieve images orvideo segments from such databases.

As is also known, a content based image retrieval system is an imageretrieval system which classifies, detects and retrieves images fromdigital libraries by utilizing directly the content of the image.Content based image processing systems refer to systems which processinformation in an image by classifying or otherwise identifying subjectmatter within the image. Such systems may be used in a variety ofapplications including, but not limited to, art gallery and museummanagement, architectural image and design, interior, design, remotesensing and management of earth resources, geographic informationsystems, scientific database management, weather forecasting, retailing,fabric and fashion design, trademark and copyright database management,law enforcement and criminal investigation and picture archiving,communication systems and inspection systems including circuitinspection systems.

Conventional content based image/video retrieval systems utilize imagesor video frames which have been supplemented with text corresponding toexplanatory notes or key words associated with the images. A userretrieves desired images from an image database, for example, bysubmitting textual queries to the system using one or a combination ofthese key words. One problem with such systems is that they rely onrestricted predefined textual annotations rather than on the content ofthe still or video images in the database.

Still other systems attempt to retrieve images based on a specifiedshape. For example, to find images of a fish, such systems would beprovided with a specification of a shape of a fish. This specificationwould then be used to find images of a fish in the database. One problemwith this approach, however, is that fish do not have a standard shapeand thus the shape specification is limited to classifying oridentifying fish having the same or a very similar shape.

Still other systems classify images or video frames by using imagestatistics including color and texture. The difficulty with thesesystems is that for a given query image, even though the images locatedby the system may have the same color, textural, or other statisticalproperties as the example image, the images may not be part of the sameclass as the query image. That is, if the query image belongs to theclass of images identified as human faces, then systems which classifyimages or video frame based image statistics including color and texturemay return images which fall within the desired color and texture imagestatistics but which are not human faces.

It would thus be desirable to provide a system and technique which maybe used in a general image search and retrieval system and which allowssearching of a plurality of different types of images including but notlimited to human or animal faces, fabric patterns, symbols, logos, artgallery and museum management, architectural image and design, interiordesign, remote sensing and management of earth resources, geographicinformation systems, scientific database management, weatherforecasting, retailing, fabric and fashion design, trademark andcopyright database management, law enforcement and criminalinvestigation, picture archiving, communication systems and inspectionsystems including circuit inspection systems. It would be particularlydesirable to have the system be capable of automatically learning whichfactors are most important in searching for a particular image or for aparticular type of image.

BRIEF SUMMARY OF THE INVENTION

Method and systems for circuit board manufacture and inspection areprovided. A captured image of a circuit board is compared with a targetimage of a reference circuit board at a similar stage in manufacture.The target image and a captured image are aligned. The captured image isdivided into a plurality of image regions. A primary image region withinthe captured image is selected, and a target image region within thetarget image is selected and properties of the primary image region andthe target image region are compared.

An example of a method of manufacturing a first printed circuit board,according to the description, includes performing a manufacturingoperation on the first printed circuit board, and inspecting a result ofthe manufacturing operation by comparing a captured image of the firstprinted circuit board having the manufacturing operation applied theretowith a target image of a second printed circuit board to which themanufacturing operation has been properly applied.

The method of manufacturing the first printed circuit board can includeone or more of the following features. Inspecting the result of themanufacturing can include receiving the captured image, aligning thetarget image and the captured image, dividing the captured image into aplurality of captured image regions, where each of the captured imageregions has at least one property, dividing the target image into aplurality of target image regions, where each of the target imageregions has at least one property, selecting a primary captured imageregion comprising at least one of the plurality of captured imageregions, selecting a first target image region from the plurality oftarget image regions, and comparing one or more properties of theprimary captured image region to a corresponding one or more propertiesof the first target image region. Assigning a first score indicating adifference between the one or more properties of the primary capturedimage region and the corresponding one or more properties of the firsttarget image region selecting a next target image region, comparing oneor more properties of the primary captured image region to acorresponding one or more properties of the next target image region,and assigning a second score indicating a difference between the one ormore properties of the primary captured image region and thecorresponding one or more properties of the next target image region.

The method of manufacturing the first printed circuit board also caninclude one or more of the following additional features. For each of apredetermined number of regions in the target image, the method cancomprise repeating selecting a target image region, comparing the one ormore properties of the primary captured image region to a correspondingone or more properties of the target image region, and assigning a scoreindicating a difference between the one or more properties of theprimary captured image region and the corresponding one or moreproperties of the target image region. Finding a best match between theprimary captured image region and one of the predetermined number ofregions in the target image. Selecting a next primary captured imageregion in the captured image, selecting a target image region, comparingone or more properties of the next primary captured image region to acorresponding one or more properties of the target image region, andassigning a score indicating a difference between the one or moreproperties of the next primary captured image region and thecorresponding one or more properties of the target image region. Themanufacturing operation can correspond to a solder paste application, acomponent placement operation, a solder reflow operation, and/or asolder joint inspection operation.

An example non-transitory storage medium with computer-readableinstructions for manufacturing a first printed circuit board, accordingto the description, can include instructions comprising code forperforming a manufacturing operation on the first printed circuit board,and inspecting a result of the manufacturing operation by comparing acaptured image of the first printed circuit board having themanufacturing operation applied thereto with a target image of a secondprinted circuit board to which the manufacturing operation has beenproperly applied.

The non-transitory storage medium can include one or more of thefollowing features. Inspecting the result of the manufacturing operationcan include receiving the captured image, aligning the target image andthe captured image, dividing the captured image into a plurality ofcaptured image regions, where each of the captured image regions has atleast one property, dividing the target image into a plurality of targetimage regions, where each of the target image regions has at least oneproperty, selecting a primary captured image region comprising at leastone of the plurality of captured image regions, selecting a first targetimage region from the plurality of target image regions, and comparingone or more properties of the primary captured image region to acorresponding one or more properties of the first target image region.The instructions can comprise code for assigning a first scoreindicating a difference between the one or more properties of theprimary captured image region and the corresponding one or moreproperties of the first target image region, selecting a next targetimage region, comparing one or more properties of the primary capturedimage region to a corresponding one or more properties of the nexttarget image region, and assigning a second score indicating adifference between the one or more properties of the primary capturedimage region and the corresponding one or more properties of the nexttarget image region.

The non-transitory storage medium also can include one or more of thefollowing additional features. For each of a predetermined number ofregions in the target image, the instructions can comprise code forselecting a target image region, comparing the one or more properties ofthe primary captured image region to a corresponding one or moreproperties of the target image region, and assigning a score indicatinga difference between the one or more properties of the primary capturedimage region and the corresponding one or more properties of the targetimage region. The instructions can comprise code for finding a bestmatch between the primary captured image region and one of thepredetermined number of regions in the target image. The instructionscan comprise code for selecting a next primary captured image region inthe captured image, selecting a target image region, comparing one ormore properties of the next primary captured image region to acorresponding one or more properties of the target image region, andassigning a score indicating a difference between the one or moreproperties of the next primary captured image region and thecorresponding one or more properties of the target image region.

An example system for processing images of a first printed circuitboard, according to the description, can include a manufacturingcomponent configured to perform a manufacturing operation on the firstprinted circuit board, and an image processing component configured toinspect a result of the manufacturing operation by comparing a capturedimage of the first printed circuit board having the manufacturingoperation applied thereto with a target image of a second printedcircuit board to which the manufacturing operation has been properlyapplied.

The system for processing images of a first printed circuit board alsocan include one or more of the following features. The image processingcomponent can be configured to receive the captured image, align thetarget image and the captured image, divide the captured image into aplurality of captured image regions, where each of the captured imageregions has at least one property, divide the target image into aplurality of target image regions, where each of the target imageregions has at least one property, select a primary captured imageregion comprising at least one of the plurality of captured imageregions, select a first target image region from the plurality of targetimage regions, and compare one or more properties of the primarycaptured image region to a corresponding one or more properties of thefirst target image region. The image processing component can be furtherconfigured to assign a first score indicating a difference between theone or more properties of the primary captured image region and thecorresponding one or more properties of the first target image region,select a next target image region, compare one or more properties of theprimary captured image region to a corresponding one or more propertiesof the next target image region, and assign second a score indicating adifference between the one or more properties of the primary capturedimage region and the corresponding one or more properties of the nexttarget image region. The image processing component can be furtherconfigured to find a best match between the primary captured imageregion and one of a predetermined number of regions in the target image.

An example of a non-transitory storage medium for circuit boardinspection, according to the disclosure, includes computer-readableinstructions comprising code for aligning a target image and a selectedimage, where each of the target and selected image has at least oneproperty, and each of the target and selected image are of at least aportion of a circuit board. The instructions further comprise code fordividing the selected image into a plurality of image regions, each ofthe image regions having at least one property, and combining propertiesin predetermined ones of the plurality of image regions. Theinstructions additionally comprise code for selecting a primary imageregion within the selected image, selecting a target image region, andcomparing one or more properties of the primary image region within theselected image to a corresponding one or more properties of the targetimage region.

The example non-transitory storage medium for circuit board inspectioncan include code implementing one or more of the following features.Assigning a score indicating a difference between the one or moreproperties of the primary image region within the selected image and thecorresponding one or more properties of the target image region.Determining a match based, at least in part, on the score. Determining asatisfactory solder joint or component placement exists on the circuitboard, based, at least in part, on the match. Determining anunsatisfactory solder joint or component placement exists on the circuitboard, based, at least in part, on the match. Retrieving the targetimage or the selected image from a database.

An example of a process for comparing two images, according to thedisclosure, includes (a) aligning a target image and a selected image,each of the target and selected images having at least one property, (b)dividing the selected image into a plurality of image regions, each ofthe image regions having at least one property, (c) combining propertiesin predetermined ones of the plurality of image regions, (d) selecting aprimary image region within the selected image, (e) selecting a targetimage region, and (f) comparing one or more properties of the primaryimage region within the selected image to a corresponding one or moreproperties of the target image region.

The example process for comparing two images can include one or more ofthe following features. (g) assigning a score indicating a differencebetween the one or more properties of the primary image region withinthe selected image and the corresponding one or more properties of thetarget image region, (h) selecting a next region in the target image,(i) comparing one or more properties of the primary image region withinthe selected image to a corresponding one or more properties of the nexttarget image region, and (j) assigning a score indicating a differencebetween the one or more properties of the primary image region withinthe selected image and the corresponding one or more properties of thenext target image region. Repeating (h)-(j) for each of a predeterminednumber of regions in the target image. The best match can be determinedby the primary image region and the one of the predetermined number ofregions in the target image with a corresponding score indicating asmallest difference. Selecting a next primary image region in theselected image, and repeating items (h)-(j) for each desired primaryimage region. Either or both of the target image and the selected imagecan be of a circuit board.

An example of a system for comparing two images, according to thedisclosure, includes an input system, an image database, and an imageprocessing system coupled with the input system and the image database.The image processing system can be configured to (a) align a targetimage and a selected image, each of the target and selected images hasat least one property, the target image is retrieved from the imagedatabase, and the selected image is received from the input system, (b)divide the selected image into a plurality of image regions, each of theimage regions having at least one property, (c) combine properties inpredetermined ones of the plurality of image regions, (d) select aprimary image region within the selected image, (e) select a targetimage region, and (f) compare one or more properties of the primaryimage region within the selected image to a corresponding one or moreproperties of the target image region.

The example system for comparing two images can include one or more ofthe following features. The input system can be configured to receiveone or more images of a circuit board. At least one plug-in moduleconfigured to provide application information used by the imageprocessing system for at least on application. The at least one plug-inmodule can provide application information used by the image processingsystem to process images having at least one item from the group ofitems consisting of: a printed circuit board, a face, a scene, a fabric,and a trademark. The image processing system further can be configuredto (g) assign a score indicating a difference between the one or moreproperties of the primary image region within the selected image and thecorresponding one or more properties of the target image region, (h)select a next region in the target image, (i) compare one or moreproperties of the primary image region within the selected image to acorresponding one or more properties of the next target image region,and (j) assign a score indicating a difference between the one or moreproperties of the primary image region within the selected image and thecorresponding one or more properties of the next target image region.The image processing system further can be configured to repeat (h)-(j)for each of a predetermined number of regions in the target image. Theimage processing system further can be configured to find a best matchbetween the primary image region and one of the predetermined number ofregions in the target image, wherein the best match is determined by theprimary image region and the one of the predetermined number of regionsin the target image with a corresponding score indicating a smallestdifference. The image processing system further can be configured toselect a next primary image region in the selected image, and repeatitems (h)-(j) for each desired primary image region.

In accordance with the present invention, image processing systemincludes a search engine coupled to an image analyzer. The mage analyzerand search engine are coupled to one or more feature modules whichprovide the information necessary to describe how to optimize the imageanalyzer for a particular application. With this particular arrangement,an image processing system which can rapidly match a primary image to atarget image is provided. Each feature module defines particular regionsof an image and particular measurements to make on pixels within thedefined image region as well as the measurements to make on neighboringpixels in neighboring image regions for a given application. The featuremodules thus specify parameters and characteristics which are importantin a particular image match/search routine. The plug-in-modulescommunicate this application specific information to image analyzer. Theinformation specified by a particular feature module will vary greatlydepending upon the particular application. By using the feature modules,generic search engines and image analyzers can be used. Thus the systemcan be rapidly adapted to operate in applications as widely varying asinspection of printed circuit boards or integrated circuits to searchfor trademark images. In each application, the particular parameters andcharacteristics which are important are provided by the feature module.

It should thus be noted that the techniques of the present inventionhave applicability to a wide variety of different types of imageprocessing applications. For example, the techniques may be used inbiometric applications and systems to identify and or verify theidentity of a particular person or thing, inspection systems includinginspection and test of printed circuit boards (including withoutlimitation any type of circuit board or module or integrated circuit) inall stages of manufacture from raw boards to boards which are fullyassembled and/or fully loaded with components and sub-assemblies(including hybrids), including solder joint inspection, post pasteinspection and post placement inspection, inspection and test ofsemiconductor chips in all stages of manufacture, from wafers tofinished chips, image or video classification systems used to searchimage or video archives for a particular type of image or a particulartype of video clip, and medical image processing applications toidentify particular characteristics in an image, such as a tumor. Thusthe phrase “image processing application” or more simply “application”as used herein below refers to a wide variety of uses.

In accordance with a further aspect of the present invention, a processfor comparing two images includes the steps of (a) aligning a targetimage and a selected image, (b) dividing the selected image into aplurality of image regions, (c) collapsing properties in predeterminedimage regions, (d) selecting a primary image region, (e) selecting atarget image region, and (f) comparing one or more properties of theselected primary image region to corresponding one or more properties ofthe target image region. With this particular arrangement, a techniquefor comparing rapidly two images is provided. By selecting predeterminedfeatures and image regions to compare between the two images, the amountof time required to process the images is reduced. By combining orcollapsing features in a selected region of an image for compactrepresentation, a comparison of a relatively large amount of informationcan be accomplished rapidly.

In accordance with a still further aspect of the present invention, amethod of manufacturing a printed circuit board includes the steps of(a) performing a manufacturing operation on a printed circuit board; and(b) inspecting the result of the manufacturing operation by comparing animage of the actual operation being performed to a target image of themanufacturing operation. With this particular arrangement, theefficiency of manufacturing a printed circuit board is increased whilereducing the cost of manufacturing the printed circuit board. Themanufacturing can correspond to any one or more steps in the printedcircuit board (PCB) manufacturing process. For example, when themanufacturing process corresponds to a solder manufacturing process,then the inspection technique may be used before and/or after thepost-paste, post-placement and post-reflow operations. For example, themanufacturing process can include the inspection before and or aftersolder application, component placement, solder reflow, solder jointinspection or any other manufacturing step. By inspecting afterpredetermined steps in the manufacturing process, it is possible todetect early defects in the manufacturing process which can be correctedprior to continuing the manufacturing process. By detecting defectsearly in the manufacturing process, the expense and time associated withmanufacturing a PCB which cannot pass a final inspection test isprovided.

In one embodiment, a method and systems for processing images aredisclosed. Two images are compared using categorization codes and imageanalysis. The first image and the second image are read from a network.A first categorization code associated with the first image and a secondcategorization code associated with the second image are read. The firstand second codes are analyzed, the first and second images are compared.It is determined if the first and second images are likely to comparefavorably based, at least in part, on outcomes from the comparing andanalyzing.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing features of this invention, as well as the inventionitself, may be more fully understood from the following detaileddescription of the drawings in which:

FIG. 1 is a block diagram of an image search and retrieval system;

FIG. 1A is a diagram of a query image and a plurality of imagesresultant from submitting the query image to the image search andretrieval system of FIG. 1;

FIG. 2 is a block diagram of an image processing system including aplurality of plug-in modules;

FIGS. 3-3B are a series of-flow diagrams representing the processingperformed by the image processing system of FIGS. 1 and 2 to compare twoimages;

FIG. 3C is a plot of attributes from a query image and a target image;

FIG. 3D is a series of images illustrating an image after a divide andcollapse operation;

FIGS. 3E and 3F illustrate deformation of a region to match a targetimage;

FIGS. 4-4B are a diagrammatic view of the alignment of a primary imageand a target image;

FIG. 5 is a flow diagram showing the processing performed to generatesearch results;

FIGS. 6 and 6A are a series of images showing the steps involved incollapsing regions of an input image to generate a collapsed image to beprovided to an image search engine;

FIGS. 6B and 6C are a series of images illustrating the comparison of acollapsed core image to a collapsed database image;

FIGS. 7 and 7A are a series diagrammatical views showing a ripplefeature;

FIG. 7B is a flow diagram explaining the steps to implement a ripplefeature;

FIG. 8 is an illustration of the selection of one or more resultingimages from an image search;

FIGS. 9-9F are a series of images which illustrate how a series ofimages which illustrate how a query image may be obtained and how toemphasize one or more portions of a query image;

FIG. 10 is a flow diagram illustrating the steps to process a targetusing multiple subsections of a primary image;

FIGS. 10A-10C are a series of images illustrating the placement of aselected image region on a target image;

FIG. 11 is a flow diagram showing the steps to compute a score andprovide an output to a user;

FIG. 11A is a flow diagram showing the steps to compute an aggregatescore;

FIG. 11B is a flow diagram showing the steps to compute a score usingthe intersection method;

FIG. 12 is a flow diagram showing the steps to perform sorting by textand then by visual similarity;

FIGS. 12A-12D are a series of views illustrating textual annotationsused to locate images and then sorting the located images by visualsimilarity;

FIG. 13 is a diagrammatical side view of a circuit component and acircuit lead connected to a circuit line;

FIG. 14 is a flow diagram showing the steps to perform image processingin a circuit inspection system; and

FIG. 15 is a diagram showing the steps to manufacture a printed circuitboard.

DETAILED DESCRIPTION OF THE INVENTION

Before describing an image search and retrieval system and thetechniques associated with, some introductory concepts and terminologyare explained.

An analog or continuous parameter image such as a still photograph maybe represented as a matrix of digital values and stored in a storagedevice of a computer or other digital processing device. Thus, asdescribed herein, the matrix of digital data values are generallyreferred to as a “digital image” or more simply an “image” and may bestored in a digital data storage device, such as a memory for example,as an array of numbers representing the spatial distribution of energyat different wavelengths in a scene.

Similarly, an image sequence such as a view of a moving roller coasterfor example, may be converted to a digital video signal as is generallyknown. The digital video signal is provided from a sequence of discretedigital images or frames. Each frame may be represented as a matrix ofdigital data values which may be stored in a storage device of acomputer or other digital processing device. Thus in the case of videosignals, as described herein, a matrix of digital data values aregenerally referred to as an “image frame” or more simply an “image” or a“frame.” Each of the images in the digital video signal may be stored ina digital data storage device, such as a memory for example, as an arrayof numbers representing the spatial distribution of energy at differentwavelengths in a scene in a manner similar to the manner in which animage of a still photograph is stored.

Whether provided from a still photograph or a video sequence, each ofthe numbers in the array correspond to a digital word (e.g. an eight-bitbinary value) typically referred to as a “picture element” or a “pixel”or as “image data.” The image may be divided into a two dimensionalarray of pixels with each of the pixels represented by a digital word.

Reference is sometimes made herein to color images with only a luminancecomponent. Such images are known as gray scale images. Thus, a pixelrepresents a single sample which is located at specific spatialcoordinates in the image. It should be noted that the techniquesdescribed herein may be applied equally well to either gray scale imagesor color images.

In the case of a gray scale image, the value of each digital wordcorresponds to the intensity of the pixel and thus the image at thatparticular pixel location.

In the case of a color image, reference is sometimes made herein to eachpixel being represented by a predetermined number of bits (e.g. eightbits) which represent the color red (R bits), a predetermined number ofbits (e.g. eight bits) which represent the color green (G bits) and apredetermined number of bits (e.g. eight bits) which represent the colorblue (B-bits) using the so-called RGB color scheme in which a color andluminance value for each pixel can be computed from the RGB values.Thus, in an eight bit color RGB representation, a pixel may berepresented by a twenty-four bit digital word.

It is of course possible to use greater or fewer than eight bits foreach of the RGB values. It is also possible to represent color pixelsusing other color schemes such as a hue, saturation, brightness (HSB)scheme or a cyan, magenta, yellow, black (CMYK) scheme. It should thusbe noted that the techniques described herein are applicable to aplurality of color schemes including but not limited to the abovementioned RGB, HSB, CMYK schemes as well as the luminosity and coloraxes a & b (Lab) YUV color difference color coordinate system, theKarhunen-Loeve color coordinate system, the retinal cone colorcoordinate system and the X, Y, Z scheme.

Reference is also sometimes made herein to an image as a two-dimensionalpixel array. An example of an array size is size 512×512. One ofordinary skill in the art will of course recognize that the techniquesdescribed herein are applicable to various sizes and shapes of pixelarrays including irregularly shaped pixel arrays.

A “scene” is an image or a single representative frame of video in whichthe contents and the associated relationships within the image can beassigned a semantic meaning A still image may be represented, forexample, as a pixel array. Having 512 rows and 512 columns. An “object”is an identifiable entity in a scene in a still image or a moving ornon-moving entity in a video image. For example, a scene may correspondto an entire image while a boat might correspond to an object in thescene. Thus, a scene typically includes many objects and image regionswhile an object corresponds to a single entity within a scene.

An “image region” or more simply a “region” is a portion of an image.For example, if an image is provided as a 32×32 pixel array, a regionmay correspond to a 4×4 portion of the 32×32 pixel array.

Before describing the processing to be performed on images, it should beappreciated that, in an effort to promote clarity, reference issometimes made herein to one or more “features” or “information” in“blocks” or “regions” of an image. It should be understood that thefeatures can correspond to any or any particular characteristic of theblock including its relationship to other blocks within the same or adifferent image. Also, such image blocks or regions should be understoodas not being limited to any particular type, size or shape of a portionof an image (i.e. the block need not have a square or a rectangularshape). It should also be understood that the image need not be anyparticular type of image.

Similarly, reference is also sometimes made herein to comparison of the“features” or “information” in one of more blocks to “features” or“information” in one or more other blocks. The other blocks may be fromthe same or a different image than the first blocks. Also the processingof the blocks need not be for any specific type of image processingapplication. Rather the processing of the blocks applies in a widevariety of image processing applications.

Accordingly, those of ordinary skill in the. art will appreciate thatthe description and processing taking place on “blocks” and “regions”could equally be taking place on portions of an image having a square,rectangular, triangular, circular, or elliptical shape of any size.Likewise, the particular field in which the image processing systems andtechniques of the present invention may be used includes but is notlimited to, biometric applications and systems to identify and or verifythe identity of a particular person or thing, inspection systemsincluding inspection and test of printed circuit boards (includingwithout limitation any type of circuit board or module or integratedcircuit) in all stages of manufacture from raw boards to boards whichare fully assembled and/or fully loaded with components andsub-assemblies (including hybrids), including solder joint inspection,post paste inspection and post placement inspection, inspection and testof semiconductor chips in all stages of manufacture, from wafers tofinished chips, image or video classification systems used to searchimage or video archives for a particular type of image or a particulartype of video clip, and medical image processing applications toidentify particular characteristics in an image, such as a tumor.

Referring now to FIGS. 1 and 1A, an image search and retrieval system 10includes an image processing system 12 having an input system 14 andoutput system 16 coupled thereto. Input System 14 is here shown toinclude a graphical user interface (GUI) 14 a, a facsimile system 14 b,a camera system 14 c which may include for example a digital camera or acamera connected to a frame grabber, a scanner 14 d, a networkconnection 14 e, a video system 14N which, may include a video camerafor example, and any other means known to those of ordinary skill in theart for capturing or otherwise obtaining images and transmitting orotherwise making image data available to an image processing system.

For example, Network Connection 14 e allows input system 14 to receiveimage data from a global information network (e.g., an internet) or onintranet or any other type of local or global network. Also coupled toimage processing system 12 is an image storage device 18 which may forexample be provided as image one or more databases having stored thereina plurality of images 20 a-20N generally denoted 20. The images 20 maybe provided as still images or alternatively the images may correspondto selected frames of a video signal which are treated as still images.

Image processing system 12 also includes an image retrieval to retrieveimages from the storage device 18. Alternatively, images may be providedto image processing system 12 via the graphical user interface 14 ausing one of a number of commercially available drawing packages such asMicrosoft Paint or any similar package. Alternatively still, a camerasuch as camera 14 c or other, image capture device may be used to feedimages to the processing system 12. Thus system 12 can receive real timeor “live” camera images instead of retrieving images from a database orother storage device. Alternatively still, images may be fed to theimaging processing system 12 via the facsimile system 14 b, the scanner14 d or the like. Regardless of how images are fed to image processingsystem 12, image processing system 12 receives the images and processesthem in accordance with techniques to be described below in conjunctionwith FIGS. 2-8B.

In general overview and as shown in FIG. 1A, a query image 20 isprovided to image processing system 12 via input system 14. The imageprocessing system 12 compares the query image 20 to each of the imagesin the storage device 18 and computes a quantitative value or “score”indicative of the closeness of the match to the query image. In the casewhere the storage device is provided as an image database, theprocessing system 12 accesses the database via a database query.

Image processing system 12 provides the results of the query to outputsystem 16 in a pictorial format as a plurality of images 22 a-22 h. Eachof the result images-22 a-22 h has a quantitative value 24 a-24 hassociated therewith. The quantitative values 24 a-24 h indicates thecloseness of the match between the query image and the image stored inthe storage device 18. A quantitative value 24 closest to a value ofzero indicates the closest match while higher values indicate a matchwhich is not as close. The quantitative values allow the imageprocessing system 12 to order the result images. The relativedifferences between the quantitative values associated with each imagecan be used to cluster groups of images based upon the closeness of thematch to the query image. For instance there is a relatively largedifference between the scores associated with images 22 b and 22 c. Thisprovides an indication that images 22 a and 22 b are closer in terms ofvisual similarity to the query image 20, than the images 22 c, 22 d andbeyond. The difference in scores between 22 f and 22 g is much smallerthan the difference in scores between 22 b and 22 c. This indicates thatimages 22 f and 22 g in relation to the query image 20, are verysimilar. Thus, it should be appreciated that it is the relative valuesof the scores rather than the absolute scores can be used to order theimages found as a result of the search.

Referring now to FIG. 2, image processing system 12 includes an imageprocessor 24, a search engine 36, an image analyzer 38, and a storagedevice interface 39. Image processor 24 interfaces with one of aplurality of plug-in modules 30 a-30N generally denoted 30.

The function of the plug-in-modules 30 is to provide the informationnecessary to describe how to optimize the image analyzer 38 for aparticular application. Each of the plug-in-modules 30 define theparticular regions of an image and particular measurements to make onpixels within that image region as well as the measurements to make onneighboring pixels in neighboring image regions for a given application.For example, Module 30 a specifies parameters, constraints andcharacteristics to be used when performing trademark image searches.Module 30 b specifies, parameters, constraints and characteristics to beused when performing searches upon images of faces. Module 30 kspecifies parameters, constraints and characteristics to be used whenperforming searches of scenes (e.g., images of waterfalls, fields,etc.). Module 301 specifies parameters, constraints and characteristicsto be used when performing searches for fabrics. Module 301 specifiesparameters, constraints and characteristics to be used when performinginspection of printed circuit boards. Module 30N specifies parameters,constraints and characteristics to be used when performing searches on astream of video images. The plug-in-modules 30 thus specify parametersand characteristics which are important in a particular imagematch/search routine. The plug-in-modules communicate this applicationspecific information to image analyzer 38.

The plug-in-modules 30 may be implemented in a variety of ways. Themodules 30 themselves may implement a technique that takes as input twoor more image regions, compares them to each other, and returns scoresof how similar they are to each other. The plug-in-modules 30 maycontain a list of image attributes that are important for computingimage similarity for a particular application, such as color, luminance,and texture. The plug-in modules may also suggest how to preprocessimages before the images are sent from storage device interface 39 toimage analyzer 38.

For example, if the query image is a trademark image and it is desirableto search the storage device 18 (FIG. 1) for trademark images, then thetrademark plug-in-module 30 a describes that orientation information isimportant. It also tells the image analyzer to ignore color informationand to treat black figures on a white background and white figures on ablack background in the same manner. The trademark plug-in-module mayalso describe how to compute orientation information and how to compactthis information for more efficient processing. These techniques aremore fully described in FIGS. 6-6C.

On the other hand, if the application is face recognition and the queryimage contains a human face, then plug-in module 30 b, which correspondsto a face optimizer, would be used. The face plug-in-module 30 bcontains information describing that global configuration is importantand relative luminance or relative color information should be used inthe matching process to be robust against illumination changes.Although, not necessary, plug-in-module 30 b may implement a techniquethat describes how to delineate the boundaries of a face in an image.This technique may be used to preprocess the query and target imagesprior to analysis. Such a technique may be used to align the query andthe target image (as shown in step 40 of FIG. 3).

The scene plug-in-module 30 k emphasizes color and local and globalstructure as important attributes when comparing two scenes. The fabricoptimizer 30N utilizes a weighted combination of color, texture, localand global structure in a similarity metric which takes as input two ormore images of fabric samples. A video sequence plug-in-module can alsobe used. In the case of the video optimizer 30N, it must describe how tomatch one still image to another image and also how to take into accountthat these images are part of a time sequence. For instance, it may beimportant to track objects over time. It may also be important tocalculate the relative position, color, or other attribute from a frameat time t1 and a frame at time t2. These calculations can be madeindependently for two or more image sequences. The processed imagesequences may then be compared.

It is not necessary to have a predefined specialized plug-in-module inorder for image analyzer 38 to compare two images. The image analyzermay use a default plug-in-module 30 k, which contains generic biases tocompare two or more images of unknown content. For instance, the defaultplug-in-module 30 k utilize color, luminance, and local relativestructure. If feedback is provided to the image processor 24 and theplug-in module, the default plug-in-module 30 x can generate a list ofassociations to important image attributes. Measuring and comparingthese attributes will therefore take precedence over measuring andcomparing other attributes. In the learning step the defaultplug-in-module 30 k can also identify significant internal parametersand ranges for their values. This learning or refining procedure can beapplied to the default plug-in-module 30 k or any of the applicationspecific plug-in-modules 30 a-30 n. The learning procedure can be usedto generate a set of parameters, characteristics and constraints for anew plug-in module or to train an existing plug-in-module to utilize aset of parameters, characteristics and constraints for a newapplication.

Many of the applications described are image search applications. Thebasic premise is that one or more images are provided to the imageprocessor 24 as positive or negative examples or samples. Together theseexamples define a semantic class of images. The task of the imageprocessor 24 is to search one or more images (which may be stored in oneor more databases) to find other images in this class. Search engine 38can be optimized to perform searches rapidly. In its most generic form,the search engine 38 is used to retrieve images from a storage deviceand to channel those images sequentially or in batches to the imageanalyzer. Images from the search engine 36, which are identified by theimage analyzer 38 or by a user may be used to refine a search or toperform a new search. This is further explained in conjunction withFIGS. 7-12 below.

FIGS. 3.3B, 5, 6D, 7B, 10, 11D, 11E, 12 and 14 are a series of flowdiagrams showing the processing performed by a processing apparatuswhich may, for example, be provided as part of an image processingsystem such as that shown in FIGS. 1 and 2 to allow processing ofvarious types of images in a variety of different applications. Therectangular elements in the flow diagrams (typified by element 40 inFIG. 3) are herein denoted “processing blocks” and represent computersoftware instructions or groups of instructions. The diamond shapedelements in the flow diagrams (typified by element 54 in FIG. 3A) areherein denoted “decision blocks” and represent computer softwareinstructions or groups of instructions which affect the processing ofthe processing blocks.

Alternatively, the processing blocks represent steps performed byfunctionally equivalent circuits such as a digital signal processorcircuit or an application specific integrated circuit (ASIC). The flowdiagram does not depict the syntax of any particular programminglanguage. Rather, the flow diagram illustrates the functionalinformation one of ordinary skill in the art requires to fabricatecircuits, or to generate computer software to perform the processingrequired of the particular apparatus. It should be noted that manyroutine program elements, such as initialization of loops and variablesand the use of temporary variables are not shown.

Some of the processing blocks can represent an empirical or manualprocedure or a database function while others can represent computersoftware instructions or groups of instructions. Thus, some of the stepsdescribed in the low diagram may be implemented via computer softwarewhile others may be implemented in a different manner e.g. manually, viaan empirical procedure, or via a combination of manual and empiricalprocedures.

It will be appreciated by those of ordinary skill in the art that unlessotherwise indicated herein, the particular sequence of steps describedis illustrative only and can be varied without departing from the spiritof the invention.

Referring now to FIGS. 3-3B, processing begins in step 40 where aselected or primary image is aligned with a target image. The primaryimage is the image which is to be matched to the target image. In someembodiments, the target image has been previously stored in a database(such as database 18 in FIG. 1) and is retrieved therefrom while inother embodiments the target image is not retrieved from a database.

Either after or before the alignment step, the primary and target imagesare each divided or segmented into a plurality of sub regions (or moresimply regions) or blocks as shown in step 42. It should be noted thatdivide step 42 may be performed as a so-called intelligent divide. Anintelligent division of the image is a procedure in which regions ofpixels that have similar attributes are formed into groups. One exampleis to group regions according to color. For a field scene, which has ablue sky and a green field, the intelligent divide would partition theimage into the blue and green parts. Another example of an intelligentdivide is to group pixels that belong to the same object. For instance,in an image of a printed circuit board, the intelligent divide wouldgroup image regions that belong to the same component. In an alternateembodiment, the intelligent divide can be performed on the target imagesin place of or in addition to the primary image.

In step 44, the properties in predetermined ones of the subregions orblocks of the primary image are combined or “collapsed” to provide arelatively compact representation of the primary. One particulartechnique to collapse the data in the subregions is to get the averagevalue of a particular parameter over the whole subregion. For instance,this collapsing step could be to find the average luminance of the subregion. Another particular technique for combining the block propertiesis described hereinbelow in conjunction with FIGS. 6-6D. It should benoted that step 44 is an optional step and in some embodiments it maynot be necessary or desirable to perform the collapsing step. In otherembodiments, however, to increase the speed in matching the primary andtarget images, it may be desirable or necessary to perform thecollapsing step. It should be rioted that divide step 42 may beperformed either before or after the collapsing step 44.

Processing then flows to step 46 in which a region or block within theprimary image (referred to herein as a primary image region) isselected. Each primary image region has associated therewith a set ofproperties or characteristics or features. In some embodiments, theprimary image region and regions within a predetermined distance of theprimary image region (referred to herein as neighbor regions) are usedin the matching process.

In the case where neighbor regions are used, the properties of both theprimary image region and the neighbor regions are used in the matchingprocess. In one embodiment, a radius R1 is selected to determine whichneighbor regions around the primary image region to include for furtherprocessing. For example, a radius of zero (i.e. R1=0) indicates that noneighbor regions have been included, where a radius greater than zero(e.g. R1=1) would indicate that at least some neighbor regions are to beincluded. The particular number of neighbor regions to include should beselected in accordance with a variety of factors including but notlimited to the particular application, the processing speed and therelative importance of particular structure in the matching process.Thus, one reason to include neighbors in the matching process is if thestructure included in the neighbor is an important consideration in thematching process. For instance, when comparing facial images, thestructure of the face is very important. In all cases there should beregions corresponding to the eyes, nose, mouth, cheeks, etc. These mustbe configured in the correct spatial organization (e.g. the cheeksshould be on either side of the nose). The magnitude of the radius R1indicates the level at which the structure is important. A radius R1having a relatively small value indicates that local structure isimportant. As the value of radius R1 increases to the limit of theheight and width of the image, then global structure is emphasized.

It should be noted that not all the neighbor regions around the primaryimage region are important. For instance, when comparing images ofsolder joints, assuming the solder joints are arranged vertically in theimage, it may be that only the regions above and below the primaryregion should be included in the calculations. It is also not necessaryfor neighbor regions to have a common boundary to the primary region.For example, one definition of neighbors could be regions that are 1region away from the primary image region.

Processing then flows to step 48 where a target image region or block isselected. In a manner similar to the selection of the primary imageregion, regions which are neighbors to the target image region can alsobe selected. In this case the neighbor regions are with respect to thetarget image region. For example, a radius R2 may be used to determinethe neighbor regions which are within a predefined distance from thetarget image region. In most cases the neighbors for the primary imageregion and the target image regions should be computed in the samefashion. If this is not the case, for instance if the radius R1 isgreater or equal to 2 (i.e. R1>=2) and radius R2 is set equal 1 (i.e.R2=1), then only the neighbor regions one step away from the primaryimage region and the target image regions should be considered in thecomputation. With R2 set to zero, a comparison between one or moreproperties of the primary image region and the target image region ismade.

Processing then flows to step 50, where one or more properties of theselected primary image region and any neighboring regions are comparedto corresponding properties in the target image region and anyneighboring regions. In step 52, a score indicating the differencebetween the primary image region and its neighbors and the target imageregions and its neighbors is computed and associated with the targetimage region. A score, for instance, could be based upon thecharacteristics of luminance and position. If the radii R1 and R2 are 0,the score can be computed as a linear combination of the absolutedifference in luminance and the absolute difference in position of theprimary and primary target image region. If radii R1 and R2 are greaterthan zero, then the score can be augmented by adding in difference inluminance and position of the corresponding primary and target regionneighbors. Often, the differences in the neighbors characteristics areweighted less than the difference in the regions themselves. Forinstance, the weights of the neighbors can be calculated by a gaussianfunction centered on the location of the primary image region. Otherweighting functions may of course also be used.

If radii R1 and R2 are greater than zero, then relative measures can bemade between the primary image region and its neighbors or the neighborsto each other. For instance, one can compute whether the neighbors havegreater, less, or equal luminance to the primary image region. The samecomputation can be made between the primary target region and itsneighbors. The difference between the primary image region and thetarget image region will be greater if order of these relativerelationships is violated in the target image region. Differences inrelative measures may be weighted and added into the total score.

Processing then flows to decision block 54 where it is determinedwhether it is necessary to compare the properties of the primary imageregion to the properties of a next target primary image region. If it isnecessary to compare the primary image region to a next target imageregion, then processing flows to step 56, where a next target imageregion is selected and then to step 58, where the primary image regionis aligned with the next target primary region for purposes ofcomparison. Processing then returns to step 48. Thus, steps 54, 56 and58 implement a loop in which the primary image region is moved over apredetermined number of target primary image regions. In some cases, theprimary image region may be moved over or compared to all image regionsin the target image. In other cases, the primary image region may bemoved only over a selected number of image regions in the target image.

Once decision is made in decision step 54 not to move to the next targetimage primary image, then processing flows to step 58 in which the bestmatch is found between the primary image region and a target imageprimary region by comparing the scores computed in step 52.

Processing then flows to step 62 where decision is made as to whethermore primary image regions should be processed. If more primary imageregions remain to be processed, then processing flows to step 64 wherethe next primary image region is selected. It should be noted that insome cases, each subregion of the primary image is selected as a primaryimage subregion for processing. In other cases, however, it may bedesirable or may not be necessary to utilize each image region in theprimary image. By processing fewer than all subregions in the primaryimage region, it may be possible to increase the speed with which aprimary image and a target image can be matched. After step 64,processing returns to step 48 and steps 48-62 are repeated until thereare no more primary images to process.

In decision block 62, when the decision is made that no more primaryimage regions to process exist, processing flows to step 66 in which thebest scores associated with each primary image region are combined inorder to provide a score of how much deformation was required to map theprimary image to the target image. One way to calculate the amount ofdeformation is to add together the best scores associated with each ofthe primary image regions.

Processing then flows to step 68 in which an output is provided to auser.

Referring now to FIG. 3C, a technique for computing a score by computinga deformation distance between a query image Q and a candidate or targetimage C is shown.

Reference numerals c₁-c₈ represents an image region of a target imageand each of the reference numerals q₁-q₈ represents an image region of aquery image. Thus q_(i) denotes the i^(th) image region of the queryimage Q while C_(j) denotes the j^(th) image region of the target imageC.

In this particular example, three different axes of a Cartesiancoordinate system represent three different attributes attribute 1,attribute 2 and attribute 3 of an image. Image regions c₁-c₈ and q₁₈ areplotted along the axis. A distance between the attributes in the queryimage and attributes in a candidate image are computed. The sum of thedistances between the points, c₁-c₈ and q₁-q₈ are totaled and theminimum value is selected as the distance corresponding to the bestmatch. Thus the best match is defined as the image having the leastamount of deformation with respect to the query image. The distance iscomputed as a function of the properties of the query image and theproperties of the candidate image. The function used can be, forexample, the absolute difference between the property of the i^(th)query image (denoted as prop(qi)) and the property of the j^(th)candidate image (denoted as prop(cj)) multiplied by a weight value.

Referring now to FIGS. 3D-3F, an image 20′ after the divide and claspsteps are performed is shown as an image 21. The primary image regionand neighbor regions are shown as image region 23. Thus image region 23is provided from primary image block 23 a and neighbor blocks 23 b-23 p.Each of the lines 69 signify relative relationships between regions. Itshould be noted that here relationships between the primary image block23 a and the neighbors 23 b-23 b-23 d, 23 e, 23 f, 23 i and 23 m areshown but it is also possible to define relationships between twoneighbor blocks (e.g. blocks 23 c and 23 d).

In FIGS. 3E and 3F a primary image region 23′ includes primary imageregion block 23 a′ and neighbors 23 b′, 23 c′ in an image are shownbeing deformed in to find a match between the primary image and a targetimage. It should be noted that the shape of the primary region in FIGS.3E and 3F is not the same as the primary image region shown in FIG. 3D.In FIGS. 3E and 3F a primary image region 23′ is shown being movedbetween two different locations on a target image 71. The deformation ofthe image region 23′ is indicated by a spring 25. The spring 25 startswhere the primary region exists in the primary image. To find a goodmatch, the image region is moved from that spot over the target image71. The spring 25 provides a visual indication of the amount by whichthe primary image is deformed to match the target image. The more thespring 25 is stretched to find a good match in the target image, themore the deformation. The spring may be implemented in a variety ofways. For example the spring may be provided a function which itselfincludes a function of distance moved and other properties. The springtechnique may be implemented by making a distance measurement andassigning to it a weight value. The weight is selected in accordancewith a variety of factors including but not limited to the emphasiswhich should be assigned one parameter (e.g. a distance parameter) vs.another different parameter (e.g. a color parameter). Thus, if it isdesired to give greater consideration to spatial distance than someother parameter, then the weight value is selected accordingly.

Referring now to FIGS. 4-4B, a primary image (PI) 70 has been dividedinto a plurality of rows, 70 a-70 h, and columns, 72 a-72 h. Althoughhere eight rows and columns are shown, those of ordinary skill in theart will appreciate that fewer or greater than eight rows and columnsmaybe used. Those of ordinary still will also appreciate that the numberof rows need not be equal to the number of columns. Rather, the numberof rows may be greater than or less than the number of columns.

Each block formed by the intersection of a row and column will bedenoted as 76XX where XX corresponds to an index identifier for aparticular block. For example, reference numeral 76 aa denotes the blockin the upper left-hand corner of image 70, while reference numeral 76 hhdenotes the block in the lower right hand corner of primary image 70.The block structure can be referred to as a grid.

The particular number of rows and columns to be used in any particularapplication to create image blocks may be selected in accordance with avariety of factors including but not limited to the desire to havedistinctive or important features in the image delineated by one or moreblocks. Consider a field scene in which the important regions are thosecorresponding to the sky and the field. It is undesirable to have a verycoarse grid where many important features of the image are mergedtogether. The most extreme example of this coarseness is to consider thewhole image as one block (1 row, 1 column). In this case, it would notbe possible to distinguish the sky from the field. It is alsoundesirable to have a grid where unimportant details in the image areseparated by blocks. Consider the same field scene and a fine grid whichdelineates each blade of grass. In the most extreme case the blocks aremade up of individual pixels.

Although primary image 70 is here shown having a square shape, those ofordinary skill in the art will appreciate that other shapes may also beused. Although primary image 70 has been divided into a plurality ofsquare regions of the same size, those of ordinary skill in the art willappreciate that in some applications it may be desirable to use othersized or shaped regions and a different region reference method could beused. For example, in some applications processing speed could beimproved or the complexity of particular computations could besimplified by selecting blocks having a rectangular, a circular or atriangular shape.

Target image 80 has likewise been divided into a predetermined number ofrows and a predetermined number of columns. It is most common to dividethe target image in the same manner as the primary imago. In this casetarget image 80 has the same number of rows and columns as primary image70.

The primary image 70 is aligned with a target image 80. This step can beperformed before or after dividing the images into regions or blocks.

In primary image 70, a first primary image block, 76 bg is selected.Next, a plurality of neighbors, proximate primary image block 76 bg, areselected. In this case, the neighboring blocks are 76 af, 76 ag, 76 bf,76 cf, and 76 cg. The primary image block 76 bg and the neighboringblocks 76 af, 76 ag, 76 bf, 76 cf, and 76 cg form a selected imageregion 78.

After choosing a first primary region, a target image block, isselected. Also, a neighbor region is defined around the target imageblock. In this particular example, a first target primary block, 86 gb,is identified and neighboring blocks 86 fa, 86 fb, 86 ga; 86 ha, and 86hb are also selected. Thus, a target image region 88 is defined withinthe target image.

It should be noted that in this particular example, the size and shapeof the primary image region 78 and the target region 88 have beenselected to be the same. It should be noted, however, that it is notnecessary for the size of the regions 78 and 88 to be the same. Forexample, in some embodiments it may be desirable not to identify anyblocks neighboring block 86 gb. In this case, only the properties ofregion 76 bg would be compared with the properties of region 86 gb todetermine a match.

As shown, however, one or more selected properties of each of the blockswithin region 78 are compared with like properties of the block withinregion 88.

The region 78 is then moved to a next portion of the target image. Itshould be noted that for an ideal match for primary image region 76 b,the ideal location should be region 86 bg in the target image.

It should also be noted that in some embodiments it may be desirable ornecessary to place constraints on the positions within target image 80at which region 78 may be placed. For example, as shown in phantom inFIG. 4B, it may be desirable to not allow region 78 to be located suchthat one of the neighbors falls outside of the boundary of the targetimage, as indicated by 89 in FIG. 4B.

The neighbor properties are matched when the selected region is comparedwith the region of the target image but the weighting for the neighborcomparisons may be given less weight than the primary image region. Forexample, in one embodiment, a Gaussian weighting scheme can be used.

Thus, the properties of primary image region 76 bg are compared with theproperties of target image region block 86 gb while the properties ofblocks 76 ag are compared with properties of block 86 fb. The propertiesof block 76 af are compared with the properties of block 86 fa, theproperties of 76 bf are compared with the properties of block 86 ga. Theproperties of neighbor block 76 cf are compared with the properties ofblock 86 ha and the properties of neighbor block 76 cg are compared withthe properties of blocks 86 hb.

The appropriate plug-in module (e.g., module 30 in FIG. 1) determineswhat specific comparisons and properties are performed. It should benoted that this will vary from application to application. For example,in an image processing application related to inspection of printedcircuit boards, one set of comparisons and properties may be used whilein an application such as face detection, a second, possibly totallydifferent set of properties and comparisons may be used. The plug-inmodule may specify to commit properties of a first target neighbor, 86fb, with the properties of a first primary image region neighbor, 76 ag.The plug-in module may also specify to compare the relative propertiesof blocks with the primary and target region. For instance, theplug-in-module may specify to compute a relative property (such asrelative luminance) between region 76 bg and 76 bf. This relativeproperty can be compared to the relative property between thecorresponding target blocks 86 gb and 86 ga.

If the properties are not consistent, then this factor would be takeninto account when computing a total score for the match at thatparticular location.

The aggregate score given a primary image region, a target image region,and a set of properties is computed as a function of the differencebetween the each of the actual values of the properties in the primaryand target region. A simple example of the function is function whichincreases linearly by the difference in the primary, and targetproperties.

Referring now to FIG. 5, a the processing steps to make a symmetricmeasurement between two images is shown. The method described in FIGS.3-3B computes how similar a primary image is to a target image. In FIG.4, a comparison was made between the primary image 70 and the targetimage 80 (i.e. primary image 70 was compared with target image 80).Based on this comparison, a similarity score was computed.

If image 80 is considered the primary image and image 70 is consideredthe target image then the primary image 80 can be compared with targetimage 70 using the comparison method described in FIGS. 3-3B. Based onthis comparison, a second similarity score can be computed. It ispossible that the second similarity score (computed by comparing primaryimage 80 with target image 70) will be different than the firstsimilarity score (computed by comparing primary image 70 with targetimage 80).

An example can be used to illustrate this point. Let the regions inimage 70 be all black. Let one region in the center of image 80 be blackand the rest white. Let luminance be the only property to be considered.When image 70 is compared to image 80, all the regions in image 70 willfind an exact luminance match with the middle region in image 80. Thus,the match between image 70 and image 80 in this case is 0, the bestpossible match. Now, reverse the roles of the two images. When image 70is compared to image 80, only the center black region will have a goodluminance match to regions in image 70. The rest will have a highluminance difference to all the regions in 70. The match from 80 to 70gives a low degree of similarity (or a high match score). Thus, thematch computation as it stands is not symmetric.

It is sometimes desirable to have a symmetric measurement. For instance,this will help to insure that the same images are returned as the resultof a database search with similar query images. For example, in FIG. 1Aimage 20 is used as the query image and images 22 a-22 h are the imagesfound as a result of the search. If a second search were performed usingone of the images 22 a-22 h as the query image (image 22 b for example),a user would expect many of the same images 22 a and 22 c-22 h to againappear in the results of the search. The steps shown in FIG. 5illustrate a technique for making a symmetric measurement and,therefore, helping meet such user expectations.

Processing begins in step 90 where a first image (image A) is matched toa second image (e.g., image B) and an aggregate score is computed. Theprocessing performed to match image A to image B is the processingdescribed above in conjunction with FIGS. 3-4B.

Processing then flows to step 92 in which the second image (i.e. imageB) is matched to the first image (i.e. image A) and a second aggregatescore is computed using the processing steps described above inconjunction with FIG. 34D.

Next, as shown in processing step 94, the scores from the two searchesare combined to provide a composite score. The scores may be combined,for example, by simply computing a mathematical average oralternatively, the scores may be combined using a different mathematicaltechnique which includes a weighting function or other technique toemphasize one of the scores.

One could also retain the scores for each property in steps 90 and 92 ofFIG. 5. Step 94 could average the score for each property independentlyand then combine the results into an aggregate match for image A to Band B to A.

Referring now to FIGS. 6-6C, an image 100 includes image regions 100a-100 p. Semantically, the image represents a picture of a square havingportions or lines 102 a-102 g as shown. Each of the image portions 102a-102 g which fall within a single image region are combined orcollapsed to thus represent all of the information included in eachimage portion in a compact manner within the image region. For example,image region 100 b contains image portions 102 a, 102 b. Image portions102 a, 102 b are thus combined and represented as an image portion 104 a(FIG. 3A within image region 100 b (FIG. 6A).

Likewise image portions 102 c, 102 d are combined and represented asimage portion 104 b in image region 100 h; image portions 102 e, 102 fare combined and represented as image portion 104 c in image region 100j; and image portions 102 g; 102 h are combined and represented as imageportion 1044 in image region 10.0 e. Thus image 102 may be compactly andefficiently represented as image 104 comprising image portions 104 a-104d. Image 104 is used to perform searching within an image database suchas database 18 (FIG. 1). The above process can be performed by anappropriate one of optimizers 30.

Each image within the database 18 is likewise compactly represented suchthat when two images are compared, a relatively large amount of imageinformation is contained within a relatively compact representation thusallowing rapid comparisons to be made. Also, by utilizing the “collapsedimage” approach, an image can be stored in a relatively small amount ofstorage space.

Referring now to FIGS. 6B, 6C a collapsed query image comprising imageportions 106 a-106 d can be compared with an exemplary collapseddatabase image 108 comprising image portions 108 a-108 f. The closenessof the match of the query image 106 to the database image 108 determinesthe value or score assigned to the database image 108 which representthe closeness of the match between the query image and the databaseimage. To compute a score, an assessment is made to determine how muchdeformation of the query image must take place to match the query image106 with the database image 108.

For example, query image portion 106 a appear is in the same imagesegment as database image portion 108 a and thus no deformation of thequery image is required to match the two image portions. Query imageportion 106 b must be deformed or moved in a downward direction by oneimage segment to thus align with database image segment lose. The samematching steps are carried out for each portion of the query imageportions 106 a-106 d with respect to each segment and portion of thedatabase image 108. Once each of the query image portions 106 a-106 dhave been tested for a match then the process is repeated for the nexttarget image which may for example be retrieved from an image database.

FIGS. 7-7B illustrate the image processing and matching techniquesdescribed above in conjunction with FIGS. 1-6D in the context of atrademark searching system. It should be appreciated, of course, thatthe concepts described in conjunction with FIGS. 7-7B can also be usedin a variety of other applications including but not limited toinspection systems, face recognition systems, scene classification andimage search systems, biotech analysis and search systems, andfabric/textile management and search systems.

Referring now to FIGS. 7-7B, in which like elements are provided havinglike reference designations, an output display screen 130 includes atool bar 132, a query image choice region 134, a selected query imageregion 136 and a results image region 138. Below each image 138 a-138 hdisplayed in the results region 138 is a corresponding value 139 a-139 hcorresponding to a measure or score indicating the similarity (or match)of each of the images 138 a-138 h to the query image 136. As can be seenin FIG. 7, image 138 a identically matches query image 136 and thusreceives a score of zero (0) indicating a best possible match betweenthe query image 136 and a target image which may, for example, be storedin a storage device such as an image database system.

It should be noted that the selected query section 136 is not limited tojust one image. The results section 138 can contain 1 or more results,not necessarily eight images as here shown. Images in the resultssection 138 can also be selected and put into the selected query region136.

In this particular example; a display of three images 134 a-134 c asquery choices, is shown. The rightmost image 134 c (the triangle withthe swirl) was chosen as the query image and placed in the querylocation 136. A search system using the techniques described above inconjunction with FIGS. 1-6D matched the query image 136 to images in animage database. A status bar 140 at the bottom of the display screenprovides a real time or pseudo-real time qualitative indication of thenumber of shows how many images the system has examined in the database.In this example, the status bar 140 is approximately ⅓ the distance.from the start point, meaning that the system has matched the queryimage to 33% of the database.

The system orders the whole database in terms of how similar eachdatabase image is to the query image. It displays the results in orderof their computed visual similarity to the query image. In FIG. 7, theeight result images with the lowest scores are shown in the resultssection 138.

In the case of FIG. 7, the system computed the similarity scores asdescribed in conjunction with FIGS. 1-6C between the query image 136 andall the images in the database.

The scores were associated with their corresponding image in thedatabase. The images in the database can thus be sorted or ordered basedon these similarity scores. These scores values are noted as 139 in FIG.7. It is not necessary, however, to use the method described inconjunction with FIGS. 1-6D compute similarity. Other algorithms may beused to compute the similarity between the query image and the images inthe database. The images in the database, thus, can be ordered based onthese different types of calculations.

As can be seen from FIG. 7, the search results of those imagesdetermined by the search system to most closely match the query imageare displayed. In this particular example, the closest eight (8)matching images from the database are displayed to a user along with thesimilarity scores. An information button 141 next to each result image138 provides a means to retrieve information related to the image. Forinstance, in this case the images correspond to trademark images. Thus,the related image information may include the name of the entity whoowns and or who filed the mark, the filing date of the mark, the date ofregistration of the mark (if any), and the class of goods with which themarked is used. In the case of an inspection system, such as a printedcircuit board inspection system, the images may correspond to circuitcomponents or printed circuit boards. In this case, the relatedinformation could correspond to information about the component type orinclude manufacturing information related to the printed circuit boardor circuit component.

It should be noted that the display may be dynamically updated during asearch. By comparing the position of images 138 a-138 h in FIG. 7 totheir positions in FIG. 7A, a so-called “ripple feature” of the searchsystem is revealed. As mentioned above, FIG. 7 showed the state of theimage search after processing 33% of the images in the database. As canbe seen from the status bar 140, FIG. 7A shows the state of the imagesearching system after processing about 66% of the images in thedatabase. By dynamically displaying the images, the top eight images areperiodically updated (e.g. after every few seconds) to show the bestimages at any one discrete instant in time.

For example, by comparing the order of the result images from 138 inFIG. 7 to the order of the result images 138 in FIG. 7A, it can be seenthat image 138 b has moved from its position in FIG. 7 as the secondbest image (after about 33% of the images to be processed in thedatabase have been processed) to a position in FIG. 7A as the fifth bestimage (after about 66% of the images in the database have beenprocessed). This indicates that three images which have been processedin the middle third of the database match the query image more closelythan image 138 b which was found in the first third of the database.

Such image updates can be provided to a user in real or pseudo realtime. A user viewing this dynamic update process in which some imagesmove from one location to another, some images move off the display, andother new images appear on the display and the effect appears as a“ripple effect.”

Referring now to FIG. 7B, a flow diagram explaining the steps requiredto implement the above described “ripple effect” are shown. It should benoted that this particular example, (as well as the description providedabove in conjunction with FIGS. 7 and 7A), assumes that a set of imagesare stored in an image database. However, the concepts described hereinabove and below can be used regardless of whether the images are storedin a database or other storage device. It should also be noted that theexample in FIGS. 7 and 7A have only one query image. The conceptsdescribed can be used with multiple query images as describedhereinbelow in conjunction with FIGS. 11-11B.

Processing begins in block 160 where a query image and a set of targetimages are selected for processing. Typically, a predetermined number oftarget images (N) are selected for processing. Processing then proceedsto block 162 in which the query image is compared with the subset of thetarget images. The comparison may be accomplished using the techniquesdescribed above in conjunction with. FIGS. 3-3D. The size of the subsetof time target images may be denoted as “x” where x is less than apredetermined number of target images N. (Note, that it is not necessaryto have “x” be the same value across all iterations of the loop162-172). A similarity score is associated with each target image in thesubset.

Processing then proceeds to block 164 in which the processed targetimages and their respective scores are stored in a list of processedimages. Next as shown in block 166, the list of processed images aresorted by their similarity score.

In block 168, the set of processed target images are moved from thedatabase of target images and the value of N is decreased. In thisexample, the number of images left to process denoted as N, is decreasedby a value corresponding to x. Processing then proceeds to block 170where the best M images are displayed to the user. Processing then flowsto decision block 172 where decision is made as to whether there are anymore target images to be processed. If there are more target images tobe processed, processing flows to block 162 and the processing in block162-170 is repeated until there are no more target images to beprocessed.

Referring now to FIG. 8, a visual display 180 of images 182 a-182 hresultant from an image search is shown. It should be noted that animage search may be interrupted or stopped before completion in whichcase images corresponding to result images at that point in time will bedisplayed.

It may be desirable to be able to stop the search and view the resultimages at times prior to the completion of the search (e.g., prior tocompletion of all images in a database) since any of the resultingimages 182 a-182 h can be selected and used as starting points for a newsearch. For example, as shown in FIG. 8, image 182 c is selected and animage processing system (e.g., system 10 in FIG. 1) can use the image182 c as the query image to perform the search.

It may also be desirable to stop the search so that any result images182 a-182 h that are interesting to the user can be stored. In FIG. 7,element 132 f corresponds to a pause button that may be used to stop theprocessing. Engaging (e.g. “clicking” or pressing or selecting the iconwith a pointer) the icon 132 f resumes the search.

Furthermore, by allowing the search to be interrupted, a user watchingthe visual display can stop the search if a match is found withoutwaiting for the image processing system to complete a search of theentire database thereby decreasing the amount of time needed to satisfya query; in this case finding a similar or infringing trademark.

It should be appreciated that although the images 182 a-182 h in FIG. 8are here shown as arbitrary shapes, the same principles apply, forexample, to processing and inspection of printer circuit boards andintegrated circuits. For example, in a printed circuit board processingsystem, the images stored in the database could correspond to images ofsatisfactory and/or unsatisfactory solder joints or componentplacements. In this case, an image of a printed circuit board (orportion thereof) being fabricated or inspected would be compared to thepreviously stored images in the database. By matching the image ofinterest to an image in the database, a decision as to whether asatisfactory/unsatisfactory solder joint or component placement existscan be made.

Referring now to FIGS. 9-9F, the manner in which a query image may beobtained is shown. The query image may be an existing image in thedatabase, as shown in FIGS. 7 and 7A. (FIG. 9 shows the query image usedto perform the search.) The query image may be scanned in from a hardcopy or captured from a camera. It may be hand drawn as shown in FIG.9A. The query image may be a manipulated version of the known image. Forinstance, FIG. 9B shows FIG. 9 rotated in the image plane clockwise 90degrees. FIG. 9C shows FIG. 9 flipped about a vertical axis. FIG. 9Dshows FIG. 9 contrast inverted. The query image may also be one of theresulting images after an initial search is performed (as discussedabove in conjunction with FIG. 8).

Referring now to FIGS. 9 and 9E-F, the manner in which one or moreportions of a query image may be emphasized is shown. FIG. 9 shows aquery image 190. Portions of this image may be emphasized according totheir importance. FIG. 9E shows two regions 192, 194 marked assignificant. The region 192 corresponds to the swirl area of the image190. This indicates that the swirl pattern of the image should be moreheavily weighted than the rest of the image for the matching process.The region 194 corresponds to the triangular area of the image 190. Thisindicates that the triangle shape at the scale of region 194 of theimage 190 should be considered to be more heavily weighted than the restof the image for the matching process.

Also a relative relationship can be defined between each of the selectedimage regions such as between regions 192, 194. An example of a relativerelationship between regions is a spatial relationship. In the case ofFIG. 9E a region similar to 192 should be “below” a region similar to194. If two regions in a target image are found to match well withregions 192 and 194 and the previously defined relative relationship issatisfied in this target image, then this thatch should be more heavilyweighted than one where the relative relationship is violated.

It should also be noted that portions of the image may be removed orerased to completely de-emphasize a feature. For instance image 196 inFIG. 9F corresponds to image 190 with the triangle portion completelyerased. Thus image 190 can be modified to provide image 196 which showsonly the “swirl” section. The image 196 can then be used as the queryimage.

Referring now to FIGS. 10 and 10A, the processing which takes place toprocess an image using a selected portion of an entire image is shown.Processing begins in step 200, where a subregion of a primary image(“primary subregion”) is selected. As shown in FIG. 10A, an entire image220 has a subregion 222 selected.

Processing then flows to decision block 202 where a decision is made asto whether the target or the primary subregion should be resized. If thedecision is made to resize one of the images (e.g. the primarysubregion), then processing flows to block 204, where the one or moreresized images are generated. It should be appreciated that in someapplications it may be desirable to resize the target image and theprimary subregions such that they are the same size, in which caseprocessing then flows to block 206 and steps performed in conjunctionwith FIGS. 3-3D are performed. In other applications, however, it may bedesirable to generate a so-called image pyramid.

Processing then flows to decision block 207 in which a decision is madeas to whether any more resized images remain to be processed. If moreimages remain to be processed then processing returns to block 205. Ifno more images remain to be processed then processing ends.

In the image pyramid case, each image in the image pyramid correspondsto the primary subregion at a different scale. In this case, each scaledprimary subregion is compared to the target image and the origin of eachof the images in the image pyramid are aligned at a particular locationwith the target image. Then processing to compare the images inaccordance with steps explained above in conjunction with FIGS. 3-3D isperformed as shown in block 206. It should be understood that the originof each of the images in the image pyramid can be placed at multipledifferent locations in the target image and a comparison between thepyramid image and the target image is made in accordance with theprocessing explained above in conjunction with FIGS. 3-3D. This is doneby proceeding through steps 208-212 in FIG. 10 for each pyramid image.The processing performed in blocks 208-212 is the same as that performedin blocks 214-218 and will be explained below in conjunction with blocks214-218.

If in decision block 202 a decision is made to not resize an image, thenprocessing flows to block 214 where the origin of the selected region isaligned with a reference point in the target image are selected.

Processing then flows to block 215 where certain conditions andconstraints are specified. For example, boundary conditions may bespecified. The boundary conditions can, for example, place limits on twopositions at which region 222 may be placed relative to the targetimage. For example, one boundary condition may be that no portion ofregion 222 may lie outside any region of the target image.

When selecting the origin, care should be taken to not select areference point which causes the subregion (e.g. region 222 in FIG. 10A)to move outside the boundaries of the target image. Such a constraintmay be useful to decrease the amount of time required to match theselected subregion to the target image because many of the possiblereference points in the target image will be invalid. Usually, theseinvalid reference points are in the top, top-right, and right portionsof the target image, forming a rotated L shape.

In some embodiments it may be desirable to specify the boundaryconditions prior to selecting a reference point. Regardless of the orderin which steps 214 and 215 are performed, the boundary conditions arecompared with the selected reference points to determine whether aselected reference point is valid.

In this particular example, subregion 222 is assumed to have an originat its upper left-most corner. It should be appreciated, however, thatother origins could also be selected for subregion 222. For example, thelowermost right hand corner of subregion 222 could also serve as theorigin. Thus, although in this example, the origin is assumed to be inthe upper left hand corner, it is not necessary for this to be so.

Once the origin of the subregion 222 and its reference point in targetimage 224 are identified, it is then possible to place subregion 222 atthat reference point in target image 224. In this particular example,the subregion 222 is shown having its origin located at the sameposition as the origin of the target image 224.

If the subregion and the target image are not the same size, it shouldbe noted that it is not necessary to place the origin of subregion 222at the origin of the target image 224. The origin of subregion 222 maybe placed at any location within target image 224 as long as all otherconstraints (including boundary conditions) are satisfied.

It should be noted that in some applications it may be desirable toselect more than one region in the primary image for matching to thetarget image. This is illustrated in FIG. 9F where regions 226, 228 areselected. In the case where two image regions are selected, in steps 214and 215, reference points in the target image for each of the subregions must be identified and constraints between properties of the twoselected image regions should be taken into account.

Processing then flows to block 216 where the processing to compare theimages in accordance with steps explained above in conjunction with FIG.3-B is performed. The data from processing with selected referencepoints is stored in a storage device (e.g. a processor memory) andprocessing moves to decision block 218 where it is determined whetheranother reference point should be selected.

If in decision block 218 a decision is made that more reference pointsremain to be processed, then processing returns to block 215. If no morereference points remain to be processed then processing ends.

The query to image processing system 12 (FIG. 1) may contain more thanone image. FIG. 11 describes a general technique to match multiple queryimages to a set of target images.

FIGS. 11A and 11B describe two methods to compute an aggregate score foreach target image and to provide output to a user.

Turning now to FIG. 11, processing begins in step 230 where a firstprimary image is selected. Processing then flows to step 232 where afirst target image is selected from a storage device. In Step 234, asimilarity score between the primary image and the target image iscomputed and a record consisting of a primary image identifier, a targetimage identifier and the score associated with the target and primaryimage is stored in a storage device which may be a temporary storagedevice (e.g. a random access memory—RAM) or a permanent storage devicesuch as a magnetic tape or compact disc read only memory (CD ROM).

Processing then flows to decision block 236 where a decision is made asto whether more target images should be processed with the same primaryimage. If the decision is made to process more target images, thenprocessing flows to block 240 where a next target image is selected anda similarity score between the primary image and the next selectedtarget image is again computed. The primary and target image identifiersand the associated score are again stored in a storage device. This loopis repeated until it is not desired to select anymore target images.

Processing then flows to decision block 238 where a decision is made asto whether more primary images should be processed. If a decision ismade to process more primary images, then processing flows to Step 242where a next primary image is processed and blocks 232 through 242 arerepeated until it is not desired to process anymore primary images.

Processing then flows to block 244 where a score is computed and anoutput is provided to a user. The particular manner in which thesescores could be computed is described below in conjunction with FIGS.11A and 11B.

FIG. 11A describes a method for calculating an average score for eachtarget image over all the primary or query images. This is one of manytechniques that may be used in step 244 of FIG. 11. In FIG. 11Aprocessing begins in processing block 246 where a first target image isselected. In block 248 all records with the target ID are retrieved fromthe storage device. The scores associated with that image are added anddivided by the number of primary images.

Processing then flows to block 250 where the aggregate score isassociated with the target image. Processing then flows to decisionblock 252 in which a decision is made as to whether more target imagesshould be processed. If more target images should be processed, thenprocessing flows to block 254 and blocks 248 and 250 are repeated. Thisloop is repeated until there are no more target images to process. Whenthe decision is made in decision block 252 that there are no more targetimages, then processing flows to block 256 in which the target imagesare sorted according to their aggregate score. Processing then flows toblock 258 in which the sorted images and associated scores are providedto the user. Such output may be provided in the form of a visualdisplay, an audio display or a printed display.

FIG. 11B describes a different approach to combining scores frommultiple query images. In FIG. 11B, processing begins in block 260 wherea first primary image is selected.

Processing then flows to block 262 where all records with the primaryimage ID are retrieved. From this set of records all target images aresorted according to the score associated with the primary image. Inblock 264, the list of sorted target images and their scores which areassociated with the primary image are stored in a storage device. Indecision block 266, a decision is made as to whether more primary imagesshould be processed. If the decision is made to process more primaryimages, then processing flows to block 268 in which a next primary imageis selected and processing then flows to blocks 262 to 264.

If in decision block 266 a decision is made to not process more primaryimages, then processing flows to block 270 in which a threshold numberis selected. In block 272, target images that are in the first Npositions in all the lists associated with the primary images areidentified. In block 274, the resulting subset target images are sortedby best score or best average score and provided to a user. Theinformation may be provided in the form of a visual display, an audiodisplay or a printed display or using any other display technique wellknown to those of ordinary skill in the art.

There are many other techniques to compute a similarity score based on amultiple set of query or primary images. For example one may match eachprimary image to each of the target images individually. A score is thenassociated with each target image and this record is put into a list.The list can be sorted based on the computed scores. The system shouldreturn a non-duplicative set of images that have the lowest individualscore to at least one primary image.

Rather than trying to combine multiple scores derived from multipleprimary image, one may calculate the common characteristics across theprimary images to produce a new “condensed” query image that embodiesthose characteristics. The matching may be performed using this newquery image and a target as described in FIGS. 3-3B. The output is asimilarity score between the condensed query image and the target.Target images may be sorted based on their respective scores.

Alternatively still one may match multiple query images to each otherbefore processing the target images. The goal is to find the systemparameters that find the most consistent, best similarity scorescalculated between query images. In other words, the goal is to find thesystem parameters that best explain the similarities between the queryimages. These system parameters and one or more of the query images maybe used to calculate a similarity score to the target images.

It should also be noted that the query may be refined using positiveexamples. The results may be calculated by one or more query images. Thesystem parameters may be refined by having the user choose images fromthe result set as “positive examples”. One way to refine the parametersis to alter the them such that the resulting measure of similarity givesthe lowest (or best) possible scores to the positive examples.

Alternatively, it should also be noted that the query may be refinedusing negative examples. The results may be calculated by one or morequery images where now the user may specify both positive examples(images that fit a particular criteria or embody the perceptual concept)and negative examples (images that do not embody the perceptualconcept). One way to incorporate positive and negative examples is toalter the system parameters such that the resulting measure ofsimilarity produces a maximal difference between the scores for thepositive examples and the negative examples. Another method is tocompute the commonalities between the positive examples and then toremove any features contained in the negative examples. A new queryimage that embodies the characteristics of the positive examples anddoes not contain the characteristics of the negative examples may bereduced and used to perform the matching.

Referring now to FIG. 12, the steps for first sorting images by text andthen by visual similarity is shown. Processing begins in block 276 inwhich a primary image is retrieved. In block 278, the text associatedwith the image is also retrieved. This text may be retrieved from storedinformation or by data input by a user. In block 280 all other imageswith at least one common associated word are retrieved. In block 282,these images are grouped into a list of target images. This completesthe retrieval by text step. The next step is to sort the list of targetimages by their visual similarity to the query or primary image. Inblock 284, a primary image is compared to each of the target images inthe list. The comparison is made using the techniques described above inconjunction with FIGS. 3-3D.

In block 286, the similarity score with each target image in the list isstored. In block 288, the list of target images is sorted by similarityscore and in block 290, an output of the sorted list is provided to auser.

Referring now to FIG. 12A an exemplary data format 292 is shown. In thisparticular example, data string 292 includes an image identificationfield 294, an image data field 296 and associated text fields 298. Theimage identifier field 294 may be provided for example as analphanumeric or a numeric identifier which indicates the position of theimage on the storage device on which the image is stored. For example,the image data may be stored on a magnetic tape, a magnetic disk, a readonly memory, a CD ROM or any other storage device well known to those ofordinary skill in the art. The image ID may or may not be related to theorder in which the image is stored on the disk and in some embodimentsthe image ID field may be omitted. Image data field 296 corresponds tothe values (e.g., digital values) which when properly interpreted resultin the display of an image. The value is made to depict a color image ora gray skill image or any other type of image representation known tothose of ordinary skill in the art.

Associated with each image data field 296 is a text field 298. The textfield 298 includes a textural description of the image provided by theimage data 296. Thus in this particular example, image 296 a isdescribed by text 298 a as being a triangle swirl or geometric object.The particular text associated with each image is typically subjectiveand descriptive of the particular image. Image 296 b is an image of anapple and thus the text 298 b associated with image 296 b is apple,fruit and food. Other associated text may also be associated with theapple such as the color red for example. In addition, the textinformation may not consist of a meaningful word. Often codes are usedto describe images, such as the numeric coding scheme used by the Patentand Trademark office to describe trademark images. An example of such acode is the text string 01.01.03 which is known to be associated with afive pointed star.

In the example in FIG. 12A, the data format 292 illustrates the fields294, 296 and 298 in a contiguous manner. In some applications, however,the information may not be stored contiguously but rather may beassociated by links from one field to the next. Thus in any storagedevice the image data may be stored in one portion of the memory and thetext field associated with that image data may be stored in a differentportion of the memory.

FIG. 12B shows the query image (an apple) while FIG. 12C shows the firsteight images in the database which had been annotated with the word“apple”. The results are shown in order, of how they appear in thedatabase (which has no bearing on how visually similar they are to thequery image). FIG. 12D shows the set of images with the annotation“apple” sorted by visual similarity to the query image (FIG. 12C). Thebest eight images are shown in FIG. 12D. It should be noticed in FIG.12C that images 315 c and 315 d, which are very similar to the queryimage 310, are in the third and fourth positions. After sorting byvisual similarity they appear in the first and second positions in FIG.12D. The reordering of 315 a and 315 b after visual similarity matchingshould also be noticed. These are less visually similar to the queryimage. Image 315 a is moved to the fourth position and image 315 b ismoved to the third position as shown in 12D. Image 315 g which is not atall visually similar to the query image is displaced entirely from thetop eight images. This image does not appear in FIG. 12D.

The use of text based search with visual similarity matching has severalgood attributes. The text based search ensures that the target imagesthat are returned are in some way related to the query image.Unfortunately. the resulting set of images may be extremely long andunsorted. Without visual similarity matching, a user would have to lookat each image to find the best match. Visual similarity matching sortsthe resulting set of images in terms of visual closeness to the queryimage. Thus, the most visual similar target images are given priority inthe list over less similar images. This last steps brings the mostsalient images to the attention of the user first. The user may use thesimilarity scores to determine when to stop looking at the list ofimages. If the scores get too high or there is a large jump in scores,this information may signal to the user that images after that point maynot have much visual similarity with the query image.

FIGS. 12-12D describe the use of text and visual similarity matching asa two stage process to compare a query image to a set of target images.The two searches, however, may be integrated. Target images may be givena score which is a combination of a text match and a visual similarityscore.

Referring now to FIG. 13, a circuit component 330, having a lead 332coupled to a circuit line 334 via a solder joint 336 is shown. Such acircuit component is placed on a printed circuit board and coupled tothe circuit line which is typically fabricated at part of the printedcircuit board via an etching or deposition technique.

The particular manner in which the circuit component 330, circuit line334, and lead 332, as well as the soldering technique used to couple thelead to the circuit line, may have an impact on the specific processingwhich takes place during an inspection process. However, during theinspection of the circuit component, different features orcharacteristics may become more or less important depending upon themanner and/or technique used to mount the circuit component, couple thecircuit component to the circuit line, and fabricate the circuit line.

In this particular example, inspection of the gull wing lead 332 andsolder joint 336 is described. In this particular example, it isrecognized that the relative brightness between predetermined regions atand proximate to the solder joint 336, can be used to verify the jointas an acceptable or an unacceptable solder joint. Specifically, a topsurface of circuit component 330 reflects a certain amount of light. Theangled portion 332 a of lead 332 reflects less light and thus appearsdarker than region 330 a. Similarly, flat lead portion 3326 appearsbrighter than lead portion 332 a and a first portion of solder joint 336a appears darker than lead portion 332 b. Solder portion 336 b appearsbrighter than region 336 a and solder region 336 c appears darker thanregion 336 b due to the angle at which the solder contacts the circuitline. Circuit line portion 334 a corresponds to a flat portion of thecircuit line 334 and thus reflects light at a different angle thansolder region 336 c and thus appears brighter than solder region 336 b.Thus, when inspecting the solder joint 336, the relative luminance ofeach of the regions can be used in the inspection process and it isthese relative luminance characteristics which can be specified in aplug-in module to thus expedite the image matching process.

Referring now to FIG. 14, the steps to process an image in a circuitcomponent inspection system are shown. Processing begins in step 340where an image is captured. Referring briefly to FIGS. 13 and 13A, theimage can be a portion of an entire printed circuit board containing acomponent and the leads from the component as illustrated in FIG. 13A(note in FIG. 13A only one lead is shown). The image would thus includea top view of circuit component 330, a top view of the lead 332, a topview of the solder joint 336, and a top view of the circuit line 334.Processing then proceeds to step 342 where the existence of the solderedcomponent is verified. This can be accomplished by simply matching thecircuit component region of the image portion of FIG. 13 a to a storedimage of a circuit component. Next, as shown in step 344, the circuitconnection points (i.e., the circuit leads 332) are identified. Theleads can be identified by matching a circuit component image to thecaptured image and knowing a priori where the leads are with respect thecircuit component.

Processing then proceeds to step 346 where each lead and solder joint isseen as a primary image. The target image may be another stored image(or images) of a good lead and solder joint or a synthetic or compositeimage of the lead and joint. The target lead/joint is processed inaccordance with the steps explained above in conjunction with FIGS. 3-3Busing the lead plug-in-module (as referred to in the discussion of FIG.13. The output value (i.e., the score) from the processing of step 346is compared with a threshold value. If the score from processing 346 isgreater than a predetermined threshold value, then the lead/solder jointis classified as a bad connection. (We are assuming that a low scorecorresponds to a good match, where a score of 0 is a perfect match).Processing then ends.

It should be noted that the threshold may have to be adjusted fordifferent types of leads. Leads may vary in many aspects such as shape,size, and pitch. The threshold value may also change depending on thegoals of the manufacturer. The manufacturer when setting up a productionline may want to find all the bad leads at the expense of some falsepositives. In this case the threshold in 348 should be biased towardsthe low side. The manufacturer can analyze where and why the bad leadsoccur and change the production process to compensate for these errors.(This will be explained more fully in the text in conjunction with FIG.14).

Alternatively, when the production process is in full swing, themanufacturer may want to bias the threshold towards the high side toaccept all good leads at the expense of missing a few bad leads.

It should be noted that in step 346, the target images could be pictures(real or composite) of bad leads. It would be ideal if this set oftarget images spanned the whole class of lead and joint defects. Itwould also be desirable, if each of the target images had informationregarding what defect it portrays. In this case, a new lead such asimage portions 332-334 in FIG. 13A are compared against the targetimages. If similarity is high (a low score) between the primary imageand any target image, then the primary image should be classified asdefective. The system should record not only the best similarity score,but also the target image that produced that score. This informationwill tell the system not only that the lead and joint are bad, but alsowhy they are bad.

It is very common for one component to have multiple leads on multiplesides. It should be noted images of one (or more) of the leads from thecomponent may act as the primary image(s). Images of the rest of theleads from that component may act as the target images. For instance, instep 346, image 337 can be treated as the primary image as discussedabove in conjunction with FIGS. 3-3B. The assumption here is that acomponent should have leads and joints that look very similar to eachother. One or more good looking leads and joints should act as theprimary images. Leads and joints from that component that have defectswill look very visually different from the primary image. If the primaryimage has a defect, it should look different than the other leads fromthe component. Leads from one component may be compared also acrossdifferent boards.

The target image does not have to contain both the lead and the joint.Just the solder joint as shown in 339 of FIG. 13A could be processed.Thus, the score value would represent the closeness of the match betweenimage 339 and the target image which is the image of a good solderjoint. Thus, a perfect match would indicate that image region 339 isalso an image of a good solder joint. If the match between the targetimage and the primary image 339 is not good, this indicates that thetarget image is not an image of a good solder joint. Note also thatviews other than the top view of a circuit component could be used asboth primary and target images.

Referring now to FIG. 15, in block 350 an unpopulated printed circuitboard having electrical circuit lines etched, deposited or otherwiseprovided thereon is provided to a solder paste station as shown in block352. The solder paste application station may be provided for example asa screen printer or any other device well known to those of ordinaryskill in the art to apply solder paste to a printed circuit board. Insome embodiments, the solder paste may be applied by hand. Regardless ofthe particular manner or technique used to apply solder paste to theprinted circuit board, the solder paste is applied to predeterminedregions of the printed circuit board. The solder paste should be appliedin a predetermined amount within a given range. Processing then flows toblock 54 in which a solder paste inspection system inspects the solderpaste applied at the predetermined regions of the printed circuit board.The solder paste inspection system captures one or more images of theregions of the printed circuit board at which solder paste has beenapplied and at which inspection is necessary. The image captured by thepaste inspection system may for example be an image of the entireprinted circuit board in which case selected image regions of theprinted circuit board image can be identified and compared to targetimages. The target images correspond to images of similar printedcircuit board regions in which the solder paste has been properlyapplied. By comparing the captured image region to the target region andperforming the imaging processing techniques described herein above inconjunction with FIGS. 3-14, a determination can be made as to whetherthe solder paste applied in block 352 was properly applied in each ofthe appropriate regions of the printed circuit board. If a decision ismade that the solder paste was not properly applied in one or more ofthe examined regions, then the printed circuit board is returned toblock 352 where the solder paste is reapplied in each of the regions inwhich it had not been properly applied in the first instance. Thus,blocks 352 and 354 are repeated until the paste inspection systemdetermines that the solder paste has been properly applied in eachappropriate region.

Processing then flows to block 356 in which the printed circuit boardwith the solder paste properly applied thereon is provided to acomponent placement station. The component placement station can includea so called pick and place machine or alternatively, the placementstation may involve manual placement of circuit components on theprinted circuit board. The decision to use automated or manual componentplacement techniques is made in accordance with a variety of factorsincluding but not limited to the complexity of the circuit component,the sensitivity of the circuit component to manual or machine handling,technical limitations of automated systems to handle circuit componentsof particular sizes and shapes and the cost effectiveness of usingautomated versus manual systems.

Once the circuit component is placed on the printed circuit board,processing moves through block 358 in which a placement inspectionstation performs an inspection of the placed circuit component. Theplacement inspection system includes an image capturing device tocapturing image of each of the circuit components of interest. Thecaptured image of the circuit component is compared to a target image ofa placed circuit component using the techniques described here and abovein conjunction with FIGS. 3-14. In response to the instructions fromplacement inspection station at block 358, processing can return toprocessing block 352 or processing block 356 depending upon the resultsof the placement inspection station in block 358.

Once determination is made in block 358 that the circuit component isproperly placed and no other defects are detected, processing flows toblock 360 in which a solder reflow station reflows the solder thuscoupling the circuit component to the printed circuit board. Solderreflow station may be provided as an automated station or as a manualstation. After solder reflow in block 360, processing flows to block 362where a placement and solder joint inspection station inspects eachcircuit component and solder joint of interest. If no defects aredetected, then processing flows to block 364 where a populated printedcircuit board is provided.

It should be noted that in the above description, the target images areall good examples of paste application, component placement, orre-flowed solder joints. It is possible to have a mixed population ofboth good and bad examples in the target images. The results of matchingthe input image to the good and bad examples will tell the systemwhether the image looks more similar to the good examples rather thanthe bad examples (or vice versa). This discrimination can be veryimportant in the decision of whether to pass or fail the image.Similarity of the primary image to a specific bad example can provideinformation regarding the type of defect. This information is extremelydesirable for purposes of tuning the production process.

Good and bad examples can be collected over time as multiple boards aresent through stages 350 to 362 in FIG. 15. These examples can be used tocreate or augment the target images for each of the three stages.

Each inspection stage (stages 354, 358 and 362) can collect dataregarding the number and type of defects found. This data can beanalyzed and fed back into the system to tune the process. For instanceat stage 354, the post-paste inspection stage may consistently reportinsufficient paste over the whole board. This may mean that not enoughpaste is being applied in step 352. The inspection system mayconsistently report insufficient paste in one area on the board,suggesting a clogged opening in the paste stencil. A similar analysismay be done for post placement and post-reflow defects. Feedbackregarding errors may be sent from any inspection station to any part ofthe process prior to that inspection stage. For instance, consistentdetection of insufficient solder in step 362 (post-reflow) may mean thatnot enough paste is being applied in step 352 (pre-reflow).

Having described the preferred embodiments of the invention, it will nowbecome apparent to one of ordinary skill in the art that otherembodiments incorporating their concepts may be used. It is felttherefore that these embodiments should not be limited to disclosedembodiments but rather should be limited only by the spirit and scope ofthe appended claims. All publications and references cited herein areexpressly incorporated herein by reference in their entirety.

1. A method of manufacturing a first printed circuit board, the methodcomprising: performing a manufacturing operation on the first printedcircuit board; and inspecting a result of the manufacturing operation bycomparing a captured image of the first printed circuit board having themanufacturing operation applied thereto with a target image of a secondprinted circuit board to which the manufacturing operation has beenproperly applied.
 2. The method of manufacturing the first printedcircuit board of claim 1, wherein inspecting the result of themanufacturing comprises: receiving the captured image; aligning thetarget image and the captured image; dividing the captured image into aplurality of captured image regions, wherein each of the captured imageregions has at least one property; dividing the target image into aplurality of target image regions, wherein each of the target imageregions has at least one property; selecting a primary captured imageregion comprising at least one of the plurality of captured imageregions; selecting a first target image region from the plurality oftarget image regions; and comparing one or more properties of theprimary captured image region to a corresponding one or more propertiesof the first target image region.
 3. The method of manufacturing thefirst printed circuit board of claim 2, further comprising: assigning afirst score indicating a difference between the one or more propertiesof the primary captured image region and the corresponding one or moreproperties of the first target image region; selecting a next targetimage region; comparing one or more properties of the primary capturedimage region to a corresponding one or more properties of the nexttarget image region; and assigning a second score indicating adifference between the one or more properties of the primary capturedimage region and the corresponding one or more properties of the nexttarget image region.
 4. The method of manufacturing the first printedcircuit board of claim 3, wherein for each of a predetermined number ofregions in the target image, the method further comprises repeating:selecting a target image region; comparing the one or more properties ofthe primary captured image region to a corresponding one or moreproperties of the target image region; and assigning a score indicatinga difference between the one or more properties of the primary capturedimage region and the corresponding one or more properties of the targetimage region.
 5. The method of manufacturing the first printed circuitboard of claim 4, further comprising finding a best match between theprimary captured image region and one of the predetermined number ofregions in the target image.
 6. The method of manufacturing the firstprinted circuit board of claim 5, further comprising: selecting a nextprimary captured image region in the captured image; selecting a targetimage region; comparing one or more properties of the next primarycaptured image region to a corresponding one or more properties of thetarget image region; and assigning a score indicating a differencebetween the one or more properties of the next primary captured imageregion and the corresponding one or more properties of the target imageregion.
 7. The method of manufacturing the first printed circuit boardof claim 2, wherein the manufacturing operation corresponds to a solderpaste application.
 8. The method of manufacturing the first printedcircuit board of claim 2, wherein said manufacturing operationcorresponds to a component placement operation.
 9. The method ofmanufacturing the first printed circuit board of claim 2, wherein saidmanufacturing operation corresponds to a solder reflow operation. 10.The method of manufacturing the first printed circuit board of claim 2,wherein said manufacturing operation corresponds to a solder jointinspection operation.
 11. A non-transitory storage medium withcomputer-readable instructions for manufacturing a first printed circuitboard, the instructions comprising code for: performing a manufacturingoperation on the first printed circuit board; and inspecting a result ofthe manufacturing operation by comparing a captured image of the firstprinted circuit board having the manufacturing operation applied theretowith a target image of a second printed circuit board to which themanufacturing operation has been properly applied.
 12. Thenon-transitory storage medium with the computer-readable instructionsfor manufacturing the first printed circuit board of claim 11, whereinthe inspecting the result of the manufacturing operation includes:receiving the captured image; aligning the target image and the capturedimage; dividing the captured image into a plurality of captured imageregions, wherein each of the captured image regions has at least oneproperty; dividing the target image into a plurality of target imageregions, wherein each of the target image regions has at least oneproperty; selecting a primary captured image region comprising at leastone of the plurality of captured image regions; selecting a first targetimage region from the plurality of target image regions; and comparingone or more properties of the primary captured image region to acorresponding one or more properties of the first target image region.13. The non-transitory storage medium with the computer-readableinstructions for manufacturing the first printed circuit board of claim12, wherein the instructions further comprise code for: assigning afirst score indicating a difference between the one or more propertiesof the primary captured image region and the corresponding one or moreproperties of the first target image region; selecting a next targetimage region; comparing one or more properties of the primary capturedimage region to a corresponding one or more properties of the nexttarget image region; and assigning a second score indicating adifference between the one or more properties of the primary capturedimage region and the corresponding one or more properties of the nexttarget image region.
 14. The non-transitory storage medium with thecomputer-readable instructions for manufacturing the first printedcircuit board of claim 13, wherein for each of a predetermined number ofregions in the target image, the instructions comprise code for:selecting a target image region; comparing the one or more properties ofthe primary captured image region to a corresponding one or moreproperties of the target image region; and assigning a score indicatinga difference between the one or more properties of the primary capturedimage region and the corresponding one or more properties of the targetimage region.
 15. The non-transitory storage medium with thecomputer-readable instructions for manufacturing the first printedcircuit board of claim 14, wherein the instructions further comprisecode for finding a best match between the primary captured image regionand one of the predetermined number of regions in the target image. 16.The non-transitory storage medium with the computer-readableinstructions for manufacturing the first printed circuit board of claim15, wherein the instructions further comprise code for: selecting a nextprimary captured image region in the captured image; selecting a targetimage region; comparing one or more properties of the next primarycaptured image region to a corresponding one or more properties of thetarget image region; and assigning a score indicating a differencebetween the one or more properties of the next primary captured imageregion and the corresponding one or more properties of the target imageregion.
 17. A system for processing images of a first printed circuitboard, the system comprising: a manufacturing component configured toperform a manufacturing operation on the first printed circuit board;and an image processing component configured to inspect a result of themanufacturing operation by comparing a captured image of the firstprinted circuit board having the manufacturing operation applied theretowith a target image of a second printed circuit board to which themanufacturing operation has been properly applied.
 18. The system forprocessing images of a first printed circuit board of claim 17, whereinthe image processing component is further configured to: receive thecaptured image; align the target image and the captured image; dividethe captured image into a plurality of captured image regions, whereineach of the captured image regions has at least one property; divide thetarget image into a plurality of target image regions, wherein each ofthe target image regions has at least one property; select a primarycaptured image region comprising at least one of the plurality ofcaptured image regions; select a first target image region from theplurality of target image regions; and compare one or more properties ofthe primary captured image region to a corresponding one or moreproperties of the first target image region.
 19. The system forprocessing images of a first printed circuit board of claim 18, whereinthe image processing component is further configured to: assign a firstscore indicating a difference between the one or more properties of theprimary captured image region and the corresponding one or moreproperties of the first target image region; select a next target imageregion; compare one or more properties of the primary captured imageregion to a corresponding one or more properties of the next targetimage region; and assign second a score indicating a difference betweenthe one or more properties of the primary captured image region and thecorresponding one or more properties of the next target image region.20. The system for processing images of a first printed circuit board ofclaim 19, wherein the image processing component is further configuredto finding a best match between the primary captured image region andone of a predetermined number of regions in the target image.